Statistical Significance Testing for Natural Language Processing by Rotem Dror

Author:Rotem Dror [Dror, Rotem]
Language: eng
Format: epub
Publisher: Morgan & Claypool Publishers
Published: 2020-02-14T22:00:00+00:00

Table 5.2: POS tagging results (Case B)

The U-test states that Lample et al. [2016] is stochastically larger than Ma and Hovy [2016] with a p-value of 0:00025. This result is also consistent with the prediction of the COS approach as Lample et al. [2016] is better than Ma and Hovy [2016] both in terms of mean (larger) and standard deviation (smaller). Finally, the minimum value of the ASO method is 0, which also reflects an SO relationship.

Results: Case B We demonstrate that if the measures of mean and standard deviation from the COS approach indicate that algorithm A is better than algorithm B but stochastic dominance does not hold, then it also holds that A is almost stochastically larger than B with a small ∊ > 0. As an example case we consider the experiment where the performance of a BiLSTM POS tagger with one of two optimizers, Adam [Kingma and Ba, 2014] (3898 scores) or RMSProp [Hinton et al., 2012] (1822 scores), are compared across various hyper-parameter configurations and random seeds. The evaluation measure is word level accuracy. The COS for the two models is presented in Table 5.2.

The result of the U-test came insignificant with p-value of 0:4562. The COS approach predicts that Adam is the better optimizer as both its mean is larger and its standard deviation is smaller. When comparing between Adam and RMSProp, the ASO method returns an of 0.0159, indicating that the former is almost stochastically larger than the latter.

We note that decisions with the COS method are challenging as it potentially involves a large number of statistics (five in this analysis). Our decision here is to make the COS prediction based on the mean and the standard deviation of the score distribution, even when according to other statistics the conclusion might have been different. We consider this ambiguity an inherent limitation of the COS method.

Table 5.3: NER Results (Case C)

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.